Word Embeddings¶
Introduction¶
Word embedding is a collective term for unsupervised ML models that learn to map a set of words $w$ in a vocabulary (or phrases, stems, lemmas) to vectors of numerical values. This approach reduces the number of dimensions from the number of unique words $V$ (i.e. vocabulary) to a much lower value $N$ where dimensions are shared by all words, such that vectors $w^{\prime}$ are not orthogonal any more. In addition, the way embeddings are computed, the ML models discover patterns in the word relations (such as given a context).
Given a word vocabulary $W=\{w_i\}$, $\|\texttt{Vocabulary}\|=V$, each word is represented by a unit vector (e.g. one-hot encoding in ML) such as
$w_1=\begin{bmatrix}1 \\ 0 \\ \vdots \\ 0\end{bmatrix}$, $w_2=\begin{bmatrix}0 \\ 1 \\ \vdots \\ 0\end{bmatrix}$, $w_N=\begin{bmatrix}0 \\ 0 \\ \vdots \\ 1\end{bmatrix}$, $W\in \mathbb{R}^{V \times V}$
Then the word embedding maps this representation $w_i$, $i=1,2,\dots,V$ to another representation (through Skip-gram or CBOW modeling) where the vectors are in a smaller dimension $N$, and $N << V$, and vectors $w^{\prime}_i$, $i=1,2,\dots,V$, with $\|W^{\prime}\|=N$, and $W^{\prime}\in \mathbb{R}^{N \times V}$
Both Skip-Gram and Continuous Bag of Words (CBOW) models use a neural network architecture to model the word mapping from the original $W$ to the embedding $W^{\prime}$
Word representation represents the word in a vector space $W^{\prime}$ so that if the word vectors are close to one another means that those words are related to one other.
Recall that above representation of words also used in the Tf-Idf features matrix where each column is a word and each row is a document or a sentence.
Note: Check
- bag of words - similar meanings
- skip gram not necessarily similar meaning but proximity, nearby word
N-Gram¶
An n-gram is a contiguous sequence of n items from a given sample of text. The items can be letters, syllables, or words according to the application. The n-grams typically are collected from a text or speech corpus. When the items are words, n-grams is also called shingles.
Text similarity:
- k-shingling as a similarity measure, counting k-shingles from documents
- Jaccard similarity, $J(set_1, set_2)=\dfrac{\|set_1 \cap set_2\|}{\|set_1 \cup set_2\|}$
- MinHash, a fast approximation to the Jaccard similarity
- Locality sensitive hashing
Context¶
Define the context referring to a symmetric window centered on the target word $w_t$, containing the surrounding tokens at a distance less than some window size $\texttt{ws}$ : $C_t = \{w_k | k \in [t-\texttt{ws}, t+\texttt{ws}]\}$
Skip-Gram 
¶
Skip-gram model predicts the context words within a specific window given the current word. The input layer of the neural network model uses the current word and the output layer uses the context words. The hidden layer contains nodes matching the number of dimensions $N$.
Skip-gram learns by predicting the context for a given target word maximizing $\prod\limits_{t=1}^{T}P(C_t|w_t)$.
Example¶
"The man who passes the sentence should swing the sword." Ned Stark
Sliding window size $\texttt{ws}= 5$
| Window | Target | Context |
|---|---|---|
[The,man,who] |
the | man,who |
[The,man,who,passes] |
man | the,who,passes |
[The,man,who,passes,the] |
who | the,man,passes,the |
[man,who,passes,the,sentence] |
passes | man,who,the,sentence |
[sentence,should,swing,the,sword] |
swing | sentence,should,the,sword |
[should,swing,the,sword] |
the | should,swing,sword |
[swing,the,sword] |
sword | swing,the |
CBOW¶
Continuous Bag of Words model predicts the current word given the context words within a specific window. The input layer uses the context words and the output layer uses the current word. The hidden layer is of length $N$. CBOW is the opposite of Skip-gram.
The CBOW model tries to predict the target word given its context, maximizing the likelihood $\prod\limits_{t=1}^{T}P(w_t|C_t)$
To model Skip-gram or CBOW probabilities, a Softmax activation is used on top of the inner product between a target vector $\texttt{u}_{wt}$ and its context vector $\frac{1}{C_t}\sum_{w \in C_t}\texttt{v}_w$
Disadvantage: A limitation of word embeddings is that possible multiple meanings of a word are conflated into a single representation (unlike Wordnet which this knowledge is carried in the graph)
Disadvantages of WordNet¶
- Missing nuances. WordNet is not well-posed or direct to model the definition of the subtle difference between two entities. Practically, defining nuances is subjective. An example nuance: the words want and need have similar meanings, but need is more assertive.
- WordNet is subjective in itself as WordNet was designed by a relatively small community
- Maintaining WordNet is labor-intensive (such as adding new synsets, definitions, lemmas, etc.)
- Developing WordNet for other languages is costly
Solution: Develop a data-driven WordNet such as based on word embeddings.
Shortcomings of Bag of Words Method¶
- It ignores the order of the word, e.g., this is bad = bad is this
- It ignores the context of words. Example: "He loved books. Education is best found in books". Two vectors one for "He loved books" and other for "Education is best found in books." would be created and a direct approach could treat these two vectors as orthogonal (i.e. independent), losing the relation and context.
